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(54) Method and system for maintaining freeform ink annotations on changing views 



(57) The method and system o! the invention main- 
tains the iritfaalontftte 

Digital inkannotatjons to objects En an electron* display 
even as the view cfianges.7ta 
lea) retationstup of the freetorm cflgltaJ hk annotalksrts 
to objects fri the vtew based upon the layout reJatkxv 



shjps of the freeform digtel Ink annotations to the ob- 
jects in the view If theview changes, thelrwenticnrnain- 
tains the layout relaikmsmp of ihe freeform tfg&al ink 
annotations to the objects in the vtew based upon the 
mapping of the freeform digital ink annotations to foe 
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Description 

[0001] Thteappficattonb (Greeted to an ^ 
bcfirectedtopemTitttriglreel^ 
8 between freeforrn tfg^kft a 

[0002] Freetonnb^ annotation d paper A)oume^ 

ument work. Including (he processes of tMnkbip^ categorizing and reviewing, tn short, tresform Ink annotation has 

proven to be a powerful cornpreho 

etc) have a static layout and content The static nature of thew 
re [0003] However, the Irin&dAxaflTier^ 

using computers. Such documents are often oVnamic and, therefore, the accurate representation of the relationship 

of the annotations to objects tn the display has been Impossible because of the changing display. For example, a 

display of a word processor document changes as the docurrieri is un^ 

irato ifeuaibatktt eyrtems 
ts HypertiofcQeanetivtoViaud 

on Human Factors and Computing Systems, May 1 995. crtsjtge me o^ 

Information space. 

[0004] AfraatormifiQftallnkw^ 

oanMrmMatBrialonanElec^re 
» a connection between two areas h a olspta^ 

olsptay changes, Circle marks which surround other Ink identify areas, end freeforrn digital Ink marts which connect 

two area bourtdartes together 

and UvoO attempts to main tain the shape of ttte connecting mark. 

[0005] To assist in navigation of docurner^ Examples of such 

ss navigation systems are disclosed in Virtual Annotation System, by Haimon, et al. tn Proceedings of me IEEE 1996 

Virtual ReaKy Annua) International Symposium, pp. 239-245 (1998) and inWcfk0ets:3OThurnbnate fc* VVavBnolno 

In Virtual Environments by Ehrtns. et al. in video peoceedtnea of UIST 'ST. pp. 21-30 (1997). These systems mate ft 

ackwajdfwausertofreefyinteiad 

command before commerting. 
so pooq Cummt systems aiterr^ to mana^ 

one of three metftods (the terme \1^ 

methc4 known as the static aperce^ 

or the ccntentd the documerit change 

known as -MATE* which is disclosed in A Martina Based Interface for CoBaborative Witm by Hardock etaL.ii 
ss Proceedings of UIST "93 (1993), addresses changing content by maintaining a static annotated version, a changed 
version without annotation and logical links between these two versions. 
[0007] A second method known as me snapstrt 
which can be intfvkfuaDy annotated Exarnjtfesd this approach 

orated Design of a Multimedia Communication System, by Franrik. in Human-Computer Interface Oesign, Morgan 
40 Kaufman PubSshers, mc,, (1996). 

[OOOq The tWrdrrcrthod, known as the temporal asscciation approach, sen 
arid ante the two temporal 60M 
ctfB^ararcachamiisedmMarmieewn^ 
atat.faPraifldhra^ 

<* Organized InkandAuoio Notebook, bv SctriB etal. In Proceedings of CHI < 97. ACM Press, (1997). 
(0009] moomontioriafimertBcesthatac 

between the annotations and the associated objects are not maintained H the view changes. For example, if a user 

armo^es the text d a docwitent ano\ 

follow the reflowed portion cl me text Therefore, while them may be 

so text, the layout of the new view does not hdlcate this logical re^ 

(00101 The systems described as related art above also do not maintain any indication of logical relationships be- 
tween the annotations and the associated objects B the view changes. TTvoB does not maintain the freeforrn digital ink 
annotations as the areas move around and the display changes. Rather; Tivdi maintains only one specific type of 
. annotation: a single stmks from the boundary of one area to another. This restriction precludes the use of freeforrn 

& digital ink annotations in changing displays to label moving objects or to aid navigation. 
[0011] WhOe MATE rraintalns the ^ 

underlying text, it does not maintain the indications of these logical relationshqps as the view changes. 
[0012] FreestytecfcxartotaOcwto^ 
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[0013] Dyrtomttedoesndpemtitanr^^ 
viewBtn accord anc e wtth changes to the views. 
[0014] The mettad and system dlhslmeftto 
tations and oblects in ch»^ vls^ 

changes, toe rtKflhod and system Thus, the logjca! 

rebtfoshtp between the treelonn digital Ink annotation 

views based upon the layout relationships. 

[0015) The method and system ei»rnaJntah the Indication d 

anriotaltorts and objects moYnan^ 

exptortog an Wormation space w&h a hyperbofc browser, makes a freeform digital Ink annotation on a node of the 
tree, when the user changes the disp tay to bfttg another portion ol the tree imo focus, 
rotates arxJ scales along wfto the tree to^ 
to the node. 

(001 Q The methocteaTxl systems d this invert 

digital ink annotations and objects bi a view as the view of (he annotated objects changes. This btvanlkvi brings the 
experience of marking on paper to dynamic media, changing word-processor documents, information visualizations, 
virtual worlds, digital video and word proc es sors. 
[001 7| The system and rrrtrtodoi to 

"Objects that are logically related to the treeJorm dgfta) Ink annotations are deleted, insert oo\ copied or moved or oth* 
erwlse afieisd Trie methodand system a^ 
of the objects in the display change because the objects rrtknt scale, resize^ 

other aspect of the tsyixitprssentafion changes, regardtoss of whether these changes occur discrstety or continuously 
[0018] AddiUonaQyi another exemplary embodiment of the systems and methods of the Invention provides naviga 
txytdmarkero that make freeform 

permitted to sated a f rseform digital ink awioBtohanafttiBiyvlew 
in when trie selected f leetorm digital kik annotation was made. 
[0019] The preferred embodnieritsd 

,,A |„ 

wnerenc 

Fig. 1 is a block diagram of an embodiment d the syslem for displaying char^ 

Fig. 2A shows a display d an annotated hyperboBc browser in ac cordance with an embodiment d the present 
invention; 

Fig. 2B shows the annotated hynerbofc browser d Rg. 2A wto 

tatkraaffusted in accor dan ce wtfotte 

Fig. 3A shows a display cA an annotated document 

Fig. 38 shows the annotated document d Fig. 3A after nxxfificatad the docume^ 

to the freeform digital ink annotations in accordance wfth an embodiment d me present invention; 

Figa 4A^ show a ftowchart outNng a corM 

8 display in accordance with an embodiment d the prevent Invention; 

Figs. 5A-59 show a flowchart outlining a control routine tor rendering freeform digital ink annotations on a changed 
view in accordance with an embodiment d the present invention; and 

Fig 6 shows a few examples d different types d freeform digital ink annotations. These and other features and 
advant^esd this invention are descried in or are apparent from the (oDowing detailed descnption d the preferred 
embodiments. 

[0020] R&i shows one exemjxafyerrvxxfimemdto^ 

1 0tnctudss a processor 12 eornmtmicating with a memory 14 Oial stores the proo/sms^obcumentd and other data for 
practicing the invention. The processor 12 also oonummicates 

pen 22 via inpuUnrtpul device 24. The keyboard 18, mouse 20, pen 22 and any other interlace devices (rax shown) 
am operated by a user to comrd the operation 10. Contro) d the electronic system 10 is rat 

^edtotheiffleriaceo^vicesshovmandotesn^ 16 may be on 

a starid^lone pen computer or a tao^ 
(0021) As shown in Fig. 1, the system 10 tepfdsrabVirnplemfl^ 

I lowovor, the system 10 can also be implemented using a special purpose oomputer, a programmed microprocessor 
or microcontroller and any necessary peripheral htegrated circuit elements, an ASIC or other integrated circuit, a 
hardwired electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PUD, 
PLA, FPG A or PAL, or the like. In general, arty device on which a finite state machine capable d implementing the 
flow charts shown in Figs. 4A6B can be used to Implement me system 10. 
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[0022] AddWonaty as shewn In Fig. 1, to memory 14 Is preferably implemented using static or dynamic RAM. 

However, to memory 14 can aiso be Implemented us^ 

dn\e, a riardoYtvo. a Rash rnemory or to 

(Hutinsi potions of a single memory or physically distinct memories. 
0 [0023] Furtrier.&shouH be appreciated 

1 8, the Keyboard 1 8, the mouse 20, the pen 22 and the hputAnitput device 24 to the processor 1 2 can be wired or 

wtretess links to networks (not ehowm), Trtese networks can be locaJerea rietworks, wide area networks, intranets, the 

Internet, or any other dstrtouted processing and storage networks. 

[0C24J Fi& 2Ashowsad6ptayofavtewofarryr^^ 
10 annotation 4£Ff&2B snows an 

42hasbesn6C8JedandrrKwed6othtf 

spondswrft tochangoof thovlewof to 

between the f reef orm digital Ink arrotatxxi 42andto objects. 

(0O25) Ffe3Asrx^afirslvfcw50oladocume^ 
« shows a second view 60 of to drxument 52 after the document has been revised. Fig. 3B shows that (he freeform 

digital Ink annotations 54. 56 and 58 are ad ju s t ed h accordance with the logical relationships to the objects in the 

document 52 In accordance with an ernbodment d the invention. The content of the document S2 shown to Fig. 3A 

has been ec3ted to tosert to text % &o rnuch 

In response to these edits, the text that has been annotated at 54 now Bows across two fines and the 'crose-ouT 
20 freeform digital Ink annotation 54 flows along with the underlying text objects to continue to indicate the logical rata* 

Uonship between "cross-out* freeform digital ink ajuidutjon and the underlying text objects. The invention also reoog- 

nizes that the group of ink strokes at 58 b a rrargin rxfle and me system d 

margin note 58 in the margin to continue to Ihdxato to loo^cal relationship 

line. AdolttonaDy. to freeform 
25 anchor polhtsm 66 arid 68 to continue to faidk^ 

(MSG) An embodiment of the method and system of the present invention involves at least a three-etep process. 

The first step Is adding the freeform cfigrta) irtk annotation on objerAs in a first vte^ 

segments otlriofraefomicSgjttf 

ft anil* ^ — - AAAAbtu t i i !■ !■ M ^ ■ m m * - - - ■* - Jt-h^i * - * ■ - - — _ ^ m. _ ^ _ « - ^ - *» - - ^ m ah^t * - — ♦ ^ - ■ .. . ■ .i* 

seating or pal erw^UHnarjcinii]>gtnerTBeroTO 
90 between the freeform digital ink annotation and me corresponding objects as the display changes. 
[0027) Optional m an ertiodfoentd 

markers. For example, the user can select an annexation in a view and retiim to to 
the annotation was created. 

[0028] TrtttregfCOT digital hkermotatta relies on 

» an object model for (he view. In the example of Figs. 2A and 2$ to objects are the nodes in the hyperboSc browset 

in the example of Figs. 3A and 3B> to objects are (he text effractors. In a third example (not shown), the objects are 

to output horn an object tracking system running on a video data stream. 

[0029] /viembttflnr^dtomefo^ 

in a view by ptBdrt9 anrim to to und 
40 mark is arttrwedtotoexx resc ori dm g span 

in the view. In to example shown in Figs. 3A and 3& a comment 58 in to margfri is anchored to a position h the 

document to to margin near afi of the corresportolr^worQ^achamctera 

[0030] torn complex freeform di^ 

end 3ft consider a freeform tfpjtalirtk 
«s toanctioredtotospanciwc^cfchBfa^tam 

its end point while to line connecting to circle 68 and the tip of to arrow 66 is not endured 

[0031] An even mete cornptexfreetomtf^ 

a virtual world An example of such an amotaticn involves to 

virtual world The layout of to door changes in a view d to virtual work! 8a to 
» and system of an embodiment of to trtvention maintain to indication of to togica) relationship between the freeform 

dlgitaJ hk armotstion a»» to eta 

tatbn retting wto to door. ^ 

[0O32J For to purposes of to foUowingdsc^ 

slonaJ coortb^ to a view A stroxa is a Rst 
65 in a view A repels a portion of a two dn^ fvexampto, byabourtfngbaxct bya 

Dst ot portions. A character is a charactwtn a to* document 

text ere maintained as to text cha 

margins. Astmple anchor b a character, actaracterposta 
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anchors with offsets. An anchor b either a simple anchor or a oornpoGitaerchot An anchored Btroko ban anchor ptus 
a flat of offsets. 

[0033] An smbodrnentd the method arid system c4 

objects move. Unanchored segments d the treats 
' times adjacent anchor points move apart beca u se tha underlying objects move apart For example, if a taxi phrase 

renews ftom a single fine across two tines (as shown in Ffc 3A art 38) Jn such cases, me oc^ 

formerly acescent anchored points can be left out to reduce tiutter and any unanchored ink between those points can 

be removed. Thus, as shown in Fig, 38, the reflowed freetorm ctigEtal ink annotation Is spQ end does not show Ink 

extending between the two segmeirts, which are en two se^ 
w then Ihe freetorm digital ink annotations can be remapped to the nearest remaining objects In (ho document It aO 

anchor ob ja cta tor the freeform digital ink annotation are deleted then the annotation can be d etet eo l 

[0O34| The typed the ancrm for the freelo^ 

the kind and number of the corresponding objects. One method, d et ailed below* for computing ink anchors is to select 
the nearest object tor each segment to the Ml Aftefnaflvty 

used to determine the anchor objects. In another case, a freetorm digital ink annotation with multiple anchors may 
cause unwanted extortions what the view changes, ft avoid this problem, an en*odlmam of the present invention 
groups segments of the freeform digital ink annotation together to reduce the number of anchors. 
[0035] As briefly described earflen an ein^ 

three processes. Tto first process receives freeform tha second process computes 

2° a iop^cat mapping between objects in (he view arid 

thefreeformtfg^hkartnotationscfla 

annotations to navigate to a previous state. 

(0036) The frst process foctotasens^ 

the Ink strokes by oorutectbig m 
25 ink annotation. These Ink strokes may be grouped, tor example, by a simple time threshold The second process 

receives the freetorm dg&aJ ink annotations as biput end rrcu& to 

tations. The third process updates the view when the oocument changes. This process uses the mapping to render 
the freetorm cSgital Ink annotation on the objects as tte view changes. 
[00371 The procedures art the ernboti 
30 white the user b inking. However, the method and system of the invention Is not Bmtted to ftppflcatlnns in which the 
objects being anrrtated ob nrtc 

marks. Alternatively, the video can continue white the user marks and the state of the objects at some point in Urns 

can be used to interpret the objects and update the mapp^ between trie ob 

lions. 

35 [0038] Trie fourth ctccess takes a eeiecta 
and. aadoes, replaces the currem view win ^ 
Thb process requires storing an a dbt u on ai mapp i hgfrom 
view. 

[0O3flj The present invention recognizes 
40 other. One example is a text phrase that is buOt out of weeds. 1 1 a phrase has a freeform digital ink annotation that 

circles the phrase and if that phrase reflows across two Ones, the invention recognizes this spQ and may either reflow 

the freetorm digital Ink annotation by splitting tt up into several segments* each ot which follows the corresponding 

underlying words as (hey move, or the system may choose to draw fines between matching segments of the drcta. 

Aftsmativefy, the freeform dig&al ink annotation could be repficated at a smaller scale on each WMdual word. 
<* [0040] A user can also copy a phrase to artotherp 

entirely new and unrelated object, without the circle annotation. 

[0041) Conversely, objects can men^^ 

taled words is c^ed then the two annotated 

toforma8ir^arinotaticflOf(hem 
» [0049] Objects may also disappear entirely For example, words-in a text editor can be deleted. In this case, the 

freeform digital ink annotations associated wim toe objert can d^ 

tween the surrounolng objects. 

[00431 Objects can also chanp. For exam 

w&h a new word. The new word could be considered a a new version of the original object The annotation on the 
65 original word may then be aasoorted won me new object to 

to the new word, for example, by stretching or srutnHng, Anematrvefy, the annotation could be removed from the disptayi 
[0044] An embodiment of the method and system of this invention may group ink strokes on text to form a single 
freetorm dipjtBJ ink annotatm under- 
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tirtestaoBS-outa, Insertions, margin notes, and cortrtectora Annotations that are cfrctes are identiied by Ink stroke 
cirdes that oortah toil and tta 

cross-oid annotations are identified by elongated marks which are eflher dlrecfly on the tod a underneath the text 
These amotattons are rrapped to the 
an hsen^ Bite annotations dps 1^ 
mark appeare in the text, for example 

margin, the margin note annotations are mapped to the text that b adjacent to the vertical and/or horizontal postion 

of the annotations. Connector aruxteiione are Identified e* 

armolBlionorbyt8SSQS(le.,acfr^ 

connector portion. Connector annotations are mapped to two objects Independent^ The first object is the object thai 
ocaxrem the start o* an anrwtatlon 

be positions In the text, or they may be other annotations. As an exempts, one end ofl the One portion of a tasso is 
mapped to the drtiaannotBlk^ 
annotation. One end of the One Is rrtapp^ 
[0045] Rg. 6 shows several exemplary tyo^ 

400 IneUxfing circle (or enclosure) annotations 402 and 404. an underline annotation 406, cross-out annotations 408 

and 410, insertion annotations 41 2 e^414, a ma rpjn e^ 

rriarg^mjte 420) and an arrow conriecto 422. tt should be ivtostood that the typesrf 

shown in conjunction with the descriptions ofl various exemplary emtxx&nents herein are intended to be fflustranVe 

and not limiting to any particular embodiment/type or grammar of annotation. 

[0046] The method a nd system of this hvemlonb based upon the concept 

digfrl Ink annotations. Anchors axe offsets from objects, Freetorm digftai Ink annotations are mapped to the object 
structure of the view unotertyingr^ 
betohiefarchkal object etnjdur^ 
rt fr^I >&jn<teatMTM8arrBppihgof a 

between characters. A text ed&Of maintains Ihese charactere art For characters that 

have been dstetodt the text editor marks the charactBrs and their corrosponduig charactsr positions as deleted, but 
the text editor stffl maintains a cerrespcnderie* between the ehamcta 
characters. 

[0047] An ernbcdniemd the method art 
and presentalk* of character. The 

acters has bean detetefl can acquire the position in the view that corresponds to a character portion; acquire the 
region in the view (hat corresponds to a character, can obtain left and right margin positions that correspond to a 
character position; can acquire a rarige d character positions mat com 
treeform digital ink annotation Into a set of substrokes a tegmenta* such 1^ 
character or to a position between characters. 
[0048] Oneembofaeflidforrtetrrt 

anchored strokes. The annotation database rnay be accessed to att en ar^ 

an anchored stroke from the database or to acquire a Est of ail ot tha anchored strokes. 

[0048] flgs.4AIhrouc^4Fshcwa flowchart a 

to objects in an underlying view n accordance wim an embody 
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First, classify the group of strokes as circles, lmderiines/croswmis, insertions, margin 
notes* or connectors. Notice that this may require splitting some oftfae strokes* and 
may result in several groups of strokes. Then, 
Sort the new groups of strokes so that connectors appear after any groups they 
connect to 

For each new group of Strokes 

If the group is a circle or an underline or a erosion!, then 

Split the group of strokes into a fist of strokes and character* 
For each stroke and character pair in the list 

Create a new simple anchor corresponding to the character 
Get the region cmiesponding to the character 
Let anchor position be die center of the region 
Create a new anchored stroke based on die simple anchor 
For each position in die stroke* 

Let offset be position minus anchor position 
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Append offset to the anchored stroke* fist of offsets 
End for 

AAA anchored stroke to tftf Anno t ation dpfrfra st 
End for 

Esc if the group is an ink insertion into the text, fan 

Let character position be the character position where the ink dips into the 
text 

Gel the position concspUTwhng to the character position 
Let anchor position be that position 
For each stroke in the group 

Create a new anchored stroke based on the character position 
For each position in the stroke 

Let offset be position minus anchor position 
Append offset to the anchored strokcte list of officts 
End for 

Add anchored stroke to the annotation database 
End for 

Else if the group is a margin annotation, then 

Let character position be the first character pksffloneoiiespo^ 
vortical extent of the annotation 

Get the left and right margin positions cojrfispontfing to character position 
If the group is mostly to the left of the left nwrgm position, then 

Let margin anchor be ( character position, flag=ieft ) 

Let anchor position be the left margin poanoo 

Else 

Let margin anchor be { character position, flagrtjght ) 
Let anchor position be the right m ^ , gin position 

End if 

For each stroke in die group, 

Geate a new anchored stroke based on the margin anchor 

For each position in die stroke 

Let offset be position nanus anchor position 
Append offset to the anchored stroked lis* of offsets 

End for 

Add anchored stroke to the annotation database 
End for 

Ebe if the group is a connector, then 

For each of starting and ending position on c onn ec tor 
If position is near an annotation 

Let simple anchor be the anchor for mat annotation 
Let anchor position be the current position of the simple 

Let offcet be position minus anchor position 
.Else (position is near an object) 

Let simple anchor be the anchor for that object 
. Let anchor position be the cuirentporioOT of the object 
Utoflittbeposincmiiuniis 

End if 
End for 
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Let composite anchor be the anchors and offsets concspondingtothe 
starting and ending positions 
$ For each stroke in the group 

Create a new anchored stroke based on the composite anchor 
For each position in the stroke 

Let offset be position minus starting position 
10 Divide offset by the distance between Restarting and 

ending positions 

Rotate offset around the origin by-angle (ending position 
minus starting position) 

is Append offset to the anchored stroke 1 * list of offsets 

End for 

Add anchored stroke to die annotation database 
End for 

a, End if 

End for 

(DOSq The control routine shown Rga. 4A-4F stalls el step 8100 and oonUmies to step S102. In step SI 02, tha 
comrdrouttop^)up8lheM 

connectors. It is to be appreciated (hat Ms may require splitting some strokes and may result in several gfoups of 

strokeswThecontrcJroutto 

after am; group to wMr* they are cc^ 

inftiatesakxy that fteratlveV processes ail of 

is a tarda, sn underthe or a cross-out 0 In step S106, the control routine determines that the current annotation is a 
so drde, an uno^rto or a cross-out then the control routine jumps to step Slia tf, on (he other hand, h step S106 the 
control routine delermhes that the current annotation does not induce a circle, an underline or a cross-out then the 
cortrcJroutto continues to st^ 

insertion If.fnstep S108, the control rcrtina determines that (ha cu^ 

routine tumps to step Si 3a tf, on (he other hand the control routine determines that the current annotation is not an 
ss ink insertion, then the control routine continues to step S110, In step S110, the control routine determines 9 the a the 
current annotation is a margin annotation, if, in step S110, the control routine o^ermines that the current annotBiion 
is a margin annotation then the control routine Jumps to step S15& If, on the other hand, in step S HO, the control 
routine determines (hat the current annotation b not a margin annotation then the control routine jumps to step S1 86. 
{0051] After completing one of the processes of Figs. 484F. which wB be explained in detail below, (he control 
40 routine continues to step Si 12 where the control routine determines if more annotations need to be processed if In 
step S112. the control routine oatermErtesD^rroraanncMnra 

to step S10S. If, on the other hand, the control routine detefmlnes that no more annotations need to be p ro c es s ed , 
then the control routine continues to step S11 4 wfterecontrd to returned 
of Rgs. 4A-4F operate. 

<s {0053| A! step S11& the control routine spto to 

character. Thus, the centred routine The control rculirra (hen con- 

tinues to step S1 18, where Ihe control routine initiates a bop to fterawVely process each substroke&haracter pair in 
the current anrtotation. instep S11B, the control routine creates a new simple anchor that corresponds to the current 
character. The control routine then continues to step $120 where the control routine acquires the region in the view 

so that corresponds to the current character. The control routine then continues to step SI 22 where the contro) routine 
seta an anchor position to the center of the region in the view The control routine then continues to step S124 where 
the control routine creates a new anchored stroke th^ 

In step S126, tha control routine sets an oflset to the position from the sunstroke minus the anchor position and con- 
tinues to step SI 2B. In step S128, the control routine appends the offset to the anchored stroke's ttst of offsets and 
» continues to step S130. In step SIX, (he control routine determtos whether more positions exists in the current 
substroke, II the control routine determines in step SI 30 ftat more positions exists 91 the current substroke, then (he 
control routine returns to step S128. fl, on the other handl the control routine determines that more positions do not 
exist In the current substroke then the control routine continues to step S132. In step S132, the control routine adds 
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to anchored strata to the annotation database, and the control routine then continues to step S134. m step S134, 
the control routine determines tf more 6Ubstroke/char8Ctsr pairs exist In the current annotation, tt, In step S134, the 
control routine Qetefrnsnes that more substrofca/character pairs exist to the current annotation then the control routine 
returns to step 311a H on the other hand, the control routine determines, to step SI 34, thai no more suhstroke/ 
charBcter pairs exist In the current annotsJion ton to control toutto 
stepSitt. 

IO0S3] AlstepSiSBtooontrdnM^setoacurre^ 

dps into the text The control routine then continues to step 81 40 where the control routine acquires meposfflon 
corresporKfing to the current character posaion The control routine then continues to step 8142 where an anchor 
position is set to the current character position. The control routine then continues to step S144 where the control 
rcirtinetnftlatesalooptoprocesaa^ 

stroke based upon the current character petition as an anchor and ttecc^ 
S 146^ to ccrtrci routine sets an ctf set to 
continues to step SI 48. tn step 8148, me cctttrol routine appenfetto 
continues to step S150. In step SI 50, the control rouuradetermtoeswhe^ 
if. in step S150, the control routine o^te^ 

returns to step S146. tt, on the other hand, to step S1», me control rajihe 
to to cunenl stroke men to control ro^ 

stroke to the annotation database and me control routine conlinues to step S154. In step S154, the control routine 
determines whether more strokes exist to the current annotation. II, tn step 8154, the control routine determines that 
more strokes exist in me current annotation then the control routine returns to step S144. If, on the other hand, me 
control routine determines, In step S154, thai rtornorestrcta exists to ^ 
continues to step $112. 

[0O54J At step 8158,010 control rwittoe sets a character poslion to to 

vertical extent ot the currem oin^ 

the tofl and right margin posffioris that correspond to to 

Iho control rojihedetem^neswr^ 

routine ottermtoes that the annotation brnostfy to to left ^ 

S1S4. H t on other hano; control 

tocontrol routine jumps tostepSl6& In step $164> me control rou 

routine continues to step 8166. In step SI 66, u^ control routine sets an anchc/o 

continues to step S172. 

[0055] InstepSlo^tocontrdrouttoaseto^ 

the oontrd routine sets an anchor position to the right margin posnton end continues to step S1 72. In step S1 72, to 
corttrd routine tottsles a locp to 

a new anchored stroke based upon themajpjnandwandconlirmes 

sets an offset to the position Irom me current stroke mimis the anchor position and continues to step 81 76. tn step 
S176. the control retftoe experts the to to an^ 

S17B, to control routine ottermirtes whether more positions exist in the current stroke. II, in step 8178, to control 
rwAtodetemtirtesftalrnore if, on 

the other hand, to step 8178, the control routine detarmines mat no more positions exists to the current stroke then 
to control routine corttoues to steps SlSXUn step SI 
oatabase and continues to step Site m 

current annotation II, in step Si 82, to control routine determines that more strokes exists In the current armotalion, 
then to control routtoe rebirns to tf^ 

no more strokes exists In to current anriotation then to control routine continues to step S156 where the control 
routine returns to step 8112. 

P0561 At step S168, to control routine initiates a loop to process afl starting or endtag positions to to current 
annotation In step $186, to control routine Determines whether the current of to starting and ending posdions are 
near another armotatkxi a an ob|ect tf 

another anriotation than the controSer conlinues to step 8188 (I, on to other hand, me control routine c^rmhes, in 
step 8188, that to position Is not near another annotation then the control routine continues to step 8182. In step 
S18& to control routtoe seta a stoiptem 

step S 190, to control routine sets an anchor position as to cun^ position to si^ 
step S196 In step S192, to control routine lets a stotptoanchwbetoancriorltt 

starting or ending position and to control routto corttimtts tostep S194. tn step S194, the control routine sets an 
anchor position to to currem St 96. In step 8198, to 

control routine sets an offset to to current position minus to anchor position and continues to step S19& In step 
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$198, the control routine determines whether more starting or erring posffions exist In (to current annotation. It, in 
step S198, the control routine ctetermlrtes that more starting or ending positions exist tn the current annotation then 
the oontrot routine returns to step $186, If, on the other hand, the control routine determines, in step S198L that no 
more stajttrig or anting r^ettim 

[0057] In step S200, the control routine initiates a loop to process 
the control routine creates an anchored stroke havfr^ a comp^ 

the starting and emfrtg positions and continues to step SSSS2. In step SS02, the control routine sets an offset to the 
current pceidon in the current stroke minus the starting position and continues to step 8204. In step $204, the control 
routine cTvktes the oOeet by the distance between the starting and ending positions and continues to step 8206. In 
step $20$, the control routine rotates ffts oflset around an origin by an angle equal to the ending postlon minus the 
starting position and continues to step S20a In step S208, the control routine appends the offset to the anchored 
stratofrBrtdoteets a jidccn tln ^ 

exists in the current stroke, ft, b step 8210, the oontrt routine oetemthese^m 
strctoOten the control routine 

that no more posftfans exists In me cum^slrotomsn the control loutirte 

corttroi routine adds ttecorrpc^ 

control routine determines H more strokes exists in the current e^n^ 

that more strokes exists In me current annotation men the control routine returns to step $200, If, on the other hand, 

the control routine detennmes that 

to step S216 where Bte control routine returns to step 5112. 

[0058] Figs. 5A-5B shows a control routine lor rendering freeform digital ink eitnotatlons on a changed view The 
now chart parallels the toDowftg pseuckxade. 

Using the am&tatioa database, gd fa 
For each anchored stroke* 

Extract the anchor and the list of ofifcets 
Let stroke be a new stroke 
If anchor is a simple anchor 

Determine if the character or character posi^ 
has been deleted 
If not, then 

Let anchor position be the position of that anchor 
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For each offset, 

Let new position be offset + anchor position 

Add new position to the stroke 
End for 

End if 

Else 

Determine if the character or character positions underiyiqg the 1* and 
2* anchors has been deleted 
If both have been deleted, then 
do nothing 

Else 

Let starting position be die position of the 1* anchor, phis the 
l'oflset 

Let ending position be the position of the 2 Bd anchor, pins die 

2*offiet 

For cadi offset, 

Let new offset be offset rotated byangje(ending 

position minus starting position) 

Multiply new ofl&et by the (Estasce between die starting 

and ending positions 

Let new position be new offiet phis starting position 
Add new position to (he stroke 
End for 

End if 

End if 

Reader stroke on the view 
End for 

[OOSflJ Tho cwrtrd routine starts at step S300 and coitinuas to step S30£ In step S30£tte 
a tfet of eJl anchored atrcto from the a^ 
extracts fteanchoraandtha to dctisete^ 

(he contra* routine tete the stroka be a new stroke and conttoes to step 8308. to step 8 
mines wtether ma anclwisasin^ 
ancto then the oon&oJ rou^ oomlrtues 

$30a, that the anchor is not a simple anchor then the control routine Jumps to step S32B. lnstepS3l0 # the control 
routine determines i) the character or character posfflon has been deleted It. in step 8310, the control routine deter* 
rrtto that the character or character positm 

other hand, the control routine cttermlnes that the character or the character position has not been deleted hen the 
control routine then continues to step 8312. tn step $312, the control routine sets the anchor position to the position 
of that anchor lor the offset and cortlinues to step 8314 In step 8314, to 
c^sel from the stroke ptua to anchof o^ 

new position to the stroke and oonttoss to step S31& In step 631& thecoma 
more otTseto for Ihe 8nehw. ff, in step 

routine returns to step S314.ll, on the rtharhartd, tho oontrc* routine de^ 
the control routine jumps to step 8320 A n stBpS320,th8ccftirdranto 
tostepS322* In step S322, the control 
If. h step 8322, the ocfitrdroutirtedetem 

the control routine returns to step 8304. If, on (ha other riandl tha control routf^ 

are no more anchored strokes in the annotation database then the control routine continues to step 8324 where the 
control routine returns control to the envtronmenl tn which the control routine operates. 
[0060] in step S326, the control routine ctoterrnfnes rj the character or character posflton undertyiig the first and 
second anchors been deleted If , in step 8326, the control routine determines that both of these crusader or character 
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posfliora have been deleted foentte 
toemttrras, tnetapS326. thattta 
been deleted (hen the contrdrojjtto 

to the position ot the first anchor plus the fot offset and contlnm to step 
the efKfing posftion to the poenlon of (ho s 
S332, the control routine setsfar^ 

position minus starting posfflon and the contrd routine cortoues to step S334. to 

tipfies the new oflset by the distance between the starting and entfirtg positions and continues to step S33B. In step 
S338, the control routine sate rite new p 

In step S338, the control routine ados the new poenion to the stroto and continue 

comiolrcuthe determines whetw 

exist men the control routine rehffits to step 8332.^ 

that no rrwre offsets exists then the control routine returns to step S320. 

P0B1J Reefomtdpjalfek annotating 

mouse or other pointing device mat persists ever Urrw. A pen based Irt^ 

a pen display, a pen display tab^ 

may trier some meaning from the shapes or the morp h ologic a l 

stroke once they have been recogrtfeed 

(0062) VVhBe the entailed descrfpb^ 

method and system of the present invent may be applied to ary^ 

changing views. Electronic book arid doeum&ftfrttrfem 

w&h a pointing deviM euch as a pen and tn^ 

readers may also include other acc^iorial features wflhcwtQrrttaflca 

[00631 ttislntenOBdtobeurto^fstoodthalth^ 

ottwr content of aoccument. bicJuc£igthedcajme^ 

such as, by way of norHirfMngexamp^ 

chapter ( a portion of an image thai is IdenttRable by a consular 

as In a video, etc. 

[0064) Itshouldbeundenttoodthatach^ 
copying and movfagrf objects or tested 

a document includes renewing of objects, resizing of objects, rotation of objects, stretching d objects, distortion of 
objects, morphlrtg of objects or the like. 

[0065J Wh3e (his invention has been descrfced in conjuration with the specific embodlmenls outSned above, it is 
evident thai many alternatives, rnodiRcatloroaiKfvan^tiom 
preferred embodernentsd the invertlon as set form above w 
es may c« made without depart^ 



Claims 

1. A metrtod lor maintaining an ^ 
least one object rn a first view 

ate Iredorm digital hk annotation to 

mappng the logical rebtionshfc> of the at least one freeform digital ink annotation to the at least one object 

based upon the layout retailor^ 

In the first view; 

generating a second view; 

renden^ the at least mtreefcmc^ 

layout relationship of the at feast m freeform digital mk 

view based upon ihe mapped lop^r^^ 

least one object in the first view; 

2. Thernethc4detaim1,whereh 
daocctimerUarvJaehang^inthelayo^ 

X The method d claims 1 or 2, wherein mapping the logical rsbticftship ccjrnpnses: 
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splitting toe art lead one freeform 

spending to a portion of the annotation that to h projdntfy to en object; and 

u eating an anchor tor each of the plural &y of subdrokes based upon the layout relationship of each of Ire 
pluraffly of sunstrokes to at toast one object In Ire flrd view 

4, The method of daim 3, wharetic 

rendering tho at toast one f roetorm dlgftBl bilt araiolalion comprtoes rendering the at toast ore freeform dtgttai 

tok annexation to response to a separation of tho plurality of objects in the second view, end 

spitting the at lead one freeform cflgfial Ink annotation further comprises spotting the at least one free form 

digital ink annotation hto at toast two freeform digital ink annotations, each freeform digital Ink annotation 

cctrespcnolT^toatteastcneotftsp^ 

& The method of any of claims 1-4, wherein: 

the at toast one object comprises at toast two objects; and 

generating the second view comprises generating lha second view In response to at toast one of merging at 
least two d the a! feast two objects and re 
one new object* 

& The method of any of claims 1 *5, further comprising mapping the logical relationship based on at toast one of a 
shape of tho at toast one freefomi digital 

to the object, and a location of the at toast one freefonti digital ink annotation In (he first view. 

7. A system for rriaJntainmg an indication of a topical relationship of at toast one freeform digital ink annotation to at 
least one object to a view as the vtow changes, the indication hctucfttg a layout rotations*^ of the at toast one 
freetorm digital ink annotation to (he at least one object, the system comprbingr 

atfspJaythatGfepteysfhetsyoirt 
one ob jttc t, and 

a processor that maps the at least one freeform digital ink annotation to the at toast one object based upon 
the layout rablionsh$ erf toe at taa^ 

the dsptay (being] responsive to a change in the first view to generate a second view and to render the at 
lB8^c*terraetCTOCfigrtaJink8 

cSgftaHnk annotation and the at least ore objed in tho sec«ri vtow base d tw (horn 
c4 the at le&m freetorm dic^ftk 

a The sydemddaim 7, whereto processor 

at toast one f reelorm cfigfta) ink annotation into a ptoraiity of substrokes, each substroke corresponding to the 
portion of the freeform digital ink anrwtato that bin prcoomfty torn 

of the plurality of subdrokos based upon (he layout relationship of each of the pfurafiry of substfokes to at least 
one object in the first view. 

a The system ofctaims 7 or a wherein the processor p^nermes Ire second vim 

at toast one object being epfit into at toast two objects and Ire at toast two c4)^ being separate 

spiltthg the at least one freeform tfgjta) ink annotation into at toast two freetorm digital ink annotations, each 

freeform dip^ irik anriotBlim 

ta The system d any of claims 7-9, whereta* 

the at toast one object comprises at toast two objects; and 

the processor generates the second view in response to at least ore d at lead two d (mailed 
being rrwgedtc$e1har and at lead 

11 . The system d any d claims 7-10. wherein the processor maps Die at toast one freeform digital ink annotation 
based on at toast ore da shaped treat toad ore (reelorm 

freeform digital fek annotation to the at toad one objed and a location d the at lead one freeform digital ink 
annotation in the first view. 



14 



EP1 016883 A2 



12. The system of any tfctalra 7*11, wherein: 

the at least one freelorfn digital ink annotation includes 8) tees) two f reetomi digital ink annotations find the at 
toast one object btctudes fit (east two objects* each object corresponding to one o! the at least two freetofm 
digital Ink annotations; and 

the processor merges at least two of the at least two freefcrm digital ink annotations into a single frestofm 
(Sgftal ink annotation In response to el least two ot the at least two objects being merged. 
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